plot(): analyze distributions

Overview

The function plot() explores the distributions and statistics of the dataset. It generates a variety of visualizations and statistics which enables the user to achieve a comprehensive understanding of the column distributions and their relationships. The following describes the functionality of plot() for a given dataframe df.

  1. plot(df): plots the distribution of each column and computes dataset statistics

  2. plot(df, x): plots the distribution of column x in various ways, and computes its statistics

  3. plot(df, x, y): generates plots depicting the relationship between columns x and y

The generated plots are different for numerical and categorical columns. The following table summarizes the output for the different column types.

x

y

Output

None

None

dataset statistics, histogram or bar chart for each column

Numerical

None

column statistics, histogram, kde plot, qq-normal plot, box plot

Categorical

None

column statistics, bar chart, pie chart, word cloud, word frequencies

Numerical

Numerical

scatter plot, hexbin plot, binned box plot

Numerical

Categorical

categorical box plot, multi-line chart

Categorical

Numerical

categorical box plot, multi-line chart

Categorical

Categorical

nested bar chart, stacked bar chart, heat map

Next, we demonstrate the functionality of plot().

Load the dataset

EDAx supports Pandas and Dask dataframes. Here, we will load the well-known adult dataset into a Pandas dataframe.

[1]:
import pandas as pd
df = pd.read_csv("https://www.openml.org/data/get_csv/1595261/phpMawTba", na_values = [' ?'])

Get an overview of the dataset with plot(df)

We start by calling plot(df) which computes dataset-level statistics, a histogram for each numerical column, and a bar chart for each categorical column. The number of bins in the histogram can be specified with the parameter bins, and the number of categories in the bar chart can be specified with the parameter ngroups. If a column contains missing values, the percent of missing values is shown in the title and ignored when generating the plots.

[2]:
from edax import plot
plot(df)
[2]:
DataPrep.EDA Report

Dataset Statistics

Number of Variables 15
Number of Rows 48842
Missing Cells 6465
Missing Cells (%) 0.9%
Duplicate Rows 52
Duplicate Rows (%) 0.1%
Total Size in Memory 30.1 MB
Average Row Size in Memory 645.7 B
Variable Types
  • Numerical: 6
  • Categorical: 9

Dataset Insights

workclass has 2799 (5.73%) missing values Missing
occupation has 2809 (5.75%) missing values Missing
native-country has 857 (1.75%) missing values Missing
fnlwgt is skewed Skewed
education-num is skewed Skewed
capital-gain is skewed Skewed
capital-loss is skewed Skewed
hours-per-week is skewed Skewed
capital-gain has 44807 (91.74%) zeros Zeros
capital-loss has 46560 (95.33%) zeros Zeros

Understand a column with plot(df, x)

After getting an overview of the dataset, we can thoroughly investigate a column of interest x using plot(df, x). The output is of plot(df, x) is different for numerical and categorical columns.

When x is a numerical column, it computes column statistics, and generates a histogram, kde plot, box plot and qq-normal plot:

[3]:
plot(df, "age")
[3]:
DataPrep.EDA Report

Overview

Distinct Count74
Unique (%)0.2%
Missing0
Missing (%)0.0%
Infinite0
Infinite (%)0.0%
Memory Size763.2 KB
Mean38.6436
Minimum17
Maximum90
Zeros0
Zeros (%)0.0%
Negatives0
Negatives (%)0.0%

Quantile Statistics

Minimum17
5-th Percentile19
Q128
Median37
Q348
95-th Percentile63
Maximum90
Range73
IQR20

Descriptive Statistics

Mean38.6436
Standard Deviation13.7105
Variance187.9781
Sum1.8874e+06
Skewness0.5576
Kurtosis-0.1844
Coefficient of Variation0.3548
  1. age has 216 outliers

When x is a categorical column, it computes column statistics, and plots a bar chart and pie chart:

[4]:
plot(df, "education")
[4]:
DataPrep.EDA Report

Overview

Distinct Count16
Unique (%)0.0%
Missing0
Missing (%)0.0%
Memory Size3.5 MB

Length

Mean9.4221
Standard Deviation2.4401
Median8
Minimum4
Maximum13

Sample

1st row 11th
2nd row HS-grad
3rd row Assoc-acdm
4th row Some-college
5th row Some-college

Letter

Count366588
Lowercase Letter308287
Space Separator48842
Uppercase Letter58301
Dash Punctuation32869
Decimal Number11894
  1. The top 2 categories ( HS-grad, Some-college) take over 50%
  1. education contains many words: 48842 words

Understand the relationship between two columns with plot(df, x, y)

Next, we can explore the relationship between columns x and y using plot(df, x, y). The output depends on the types of the columns.

When x and y are both numerical columns, it generates a scatter plot, hexbin plot and box plot:

[5]:
plot(df, "age", "hours-per-week")
[5]:

When x and y are both categorical columns, it plots a nested bar chart, stacked bar chart and heat map:

[6]:
plot(df, "education", "marital-status")
[6]:

When x and y are one each of type numerical and categorical, it generates a box plot per category and a multi-line chart:

[7]:
plot(df, "age", "education")
# or plot(df, "education", "age")
[7]:
[ ]: